Methods and Systems For Vectored Data De-Duplication

ABSTRACT

The present invention is directed toward methods and systems for data de-duplication. More particularly, in various embodiments, the present invention provides systems and methods for data de-duplication that may utilize a vectoring method for data de-duplication wherein a stream of data is divided into “data sets” or blocks. For each block, a code, such as a hash or cyclic redundancy code may be calculated and stored. The first block of the set may be written normally and its address and hash can be stored and noted. Subsequent block hashes may be compared with previously written block hashes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/674,152, entitled “Methods and Systems For Vectored DataDe-Duplication,” filed Nov. 12, 2012, by George Saliba, which is in turna continuation of U.S. Pat. No. 8,332,616 (Ser. No. 13/153,688) issuedDec. 11, 2012, which is in turn a continuation of U.S. Pat. No.7,979,670 (Ser. No. 12/019,527) issued on Jul. 12, 2011.

FIELD OF THE INVENTION

The present invention relates generally to data processing systems andmore particularly, some embodiments relate to methods and systems forperforming data de-duplication.

BACKGROUND OF THE INVENTION

Vast amounts of electronic information are stored, communicated, andmanipulated by modem computer systems. Much of this vast amount ofelectronic information is duplicated. For example, duplicate or nearduplicate copies of data may be stored on a hard drive or hard drives,communicated across a communication channel, or processed using acomputer or other electronic device. This duplicated data might be usedin many different applications and on many different electronic systems.Accordingly, data de-duplication technology may impact a broad range ofapplications.

Data de-duplication is a method of reducing or eliminating redundantfiles, blocks of data, etc. In this way, a data de-duplication systemattempts to ensure that only unique data is stored, transmitted,processed, etc. Data de-duplication is also sometimes referred to ascapacity optimized protection. Additionally, data de-duplication mayaddress rapidly growing capacity needs by reducing electronicinformation storage capacity required, transmission capacity, processorcapacity, etc.

In one example of how duplicate data might exist on a computer network,an employee may email a Word® attachment to 25 co-workers. On somesystems, a copy is saved for every employee the file was sent to,increasing the capacity requirement of the file by a factor of 25. Insome cases data de-duplication technology may eliminate the redundantfiles, replacing them with “pointers” to the original data after it hasbeen confirmed that all copies are identical. This example illustratesdata de-duplication at the file level. Data de-duplication may also beimplemented based on variable size blocks of data. In other words,redundant variable sized blocks of data may be eliminated by replacingthese blocks with a pointer to another instance of a matching block ofdata.

In some cases, data duplication might occur in a data storage system.For example, archived electronic information such as electronicdocuments, files, programs, etc. exist on backup tapes, backup harddrives, and other media. In many cases a computer may store a largenumber files, which in some cases may be duplicates of the same file ordocument, slightly differing versions of the same document, etc.Accordingly, duplicates or near duplicates might exist for manydifferent types of files, including documents, graphic files, and justabout any other type of computer file.

Additionally, duplication might occur when data is communicated. Incomputer-based systems it is common for a computer to transmit one ormore files over a computer network or other communication system to, forexample, other computers in the computer network. This network may bewired, wireless, or some combination of the two. Additionally, thenetwork may use just about any computer data communication system totransmit the data.

Different types of duplication might exist. In one type, a file or filesmay be repeatedly transmitted by a computer. For example, it is commonfor data transmitted during a backup operation to be almost identical tothe data transmitted during the previous backup operation. Accordingly,a computer, computer networks, etc. might also repeatedly communicatethe same or similar data.

In another type of duplication, a duplicate or near duplicate file orfiles, such as duplicate or near duplicate document, graphic files, etc.might be stored on a computer system. In other words, multiple copies ofa file might exist, as in the emailed document example. Accordingly,different types of file de-duplication systems and methods might addressvarious types of duplication. Some types of data de-duplication systemsand methods might relate to file duplication or near duplication thatinvolves multiple copies of the same or similar files sent during thesame transmission. Other types of data de-duplication systems andmethods may relate to file duplication that involves the same or similarfiles sent during a series of transmissions. Yet other types of datade-duplication might relate to both types of file duplication or nearduplication.

Data de-duplication might include both transmission for backup and thebackup itself. For example, some data de-duplication systems maytransmit only data that has changed since a previous backup. This datamight be stored on a daily basis or perhaps a weekly basis. In somesystems these changes in the data might be what is saved, for example,on a backup drive, disc, tape, etc. For example, a backup system mightinitially transmit a “full backup” for example, all files in a directoryor series of directories, all files on a disc or on a computer, allfiles on all disks on an entire network, etc. The full backup mightsimply be any and all files that a particular user selects for backup.The data for the full backup may be transmitted and stored using variouscommunication and storage systems. After the full backup, subsequentbackups might be based on only files that have changed. These might bethe only files subsequently transmitted, stored or both. Of course, auser might also select to do a full backup from time to time after theinitial full backup.

Systems that only make full backups might be required to store a largeamount of data. This may increase the expenses associated with thesetypes of systems due to, for example, the cost of additional harddrives, tape media, data CD's or DVD's, wear on disc drives, CD or DVDdrives, tape drives, etc. Accordingly, incremental systems might be moreefficient in terms of data storage, mechanical wear on systemcomponents, etc.

In some cases, duplicate data might also be processed in other ways by acomputer system, a network of computers, etc. For example, the systemsand methods described herein might not only be applied to data storagedevices, but to data transmission devices or any other data processingdevices that deal with blocks of data that might be redundant. Forexample, in data mining and information filtering applications,duplicate or near duplicate files might be processed by the data miningor information filtering applications. In another example, an enterprisesoftware application might receive data from a wide variety of sources.These sources might vary widely in terms of formatting, quality control,or other factors that may impact the consistency or reliability of thedata. As a result, the database may contain duplicative or erroneousdata. In many cases this data may need to be “cleaned.”

“Data cleaning,” or “data clean-up,” generally refers to the handling ofmissing data or identifying data integrity violations. “Dirty data”generally refers to input data records or to particular data fields in astring of data comprising a full data record. For example, as discussedabove, anomalies may exist because data might not conform in terms ofcontent, format, or some other standard established for the database.This dirty data may need to be analyzed.

One example where dirty data may need to be analyzed involves creditcard transactions processing. Transactions may contain electronicinformation that includes data in predetermined fields. Thesepredetermined fields might contain specific information, such as, forexample, transaction amount, credit card number, identificationinformation, merchant information, date, time, etc. Various types ofdata errors may be introduced in each of the millions of credit cardtransactions are recorded each day. For example, the merchantidentifying data field for a transaction record might be tainted withinformation specific to the individual transaction. As an example,consider a data set of transactions where the merchant name fieldindicates the merchant name and additional merchant information. Thisinformation might be added by the merchants and may include a storenumber or other merchant specific information that might not be neededby the clearinghouse to authorize or settle the transaction. In somecases it might be important to clean this data to conform to a formatthat specifies the merchant name without any of the additionalinformation. In other cases, data storage space might be saved by usingone of various data de-duplication systems and methods. For example, aname used in many transactions might be saved in one data storagelocation and a pointer might be saved in other data storage locations.

There are two main types of de-duplication. These methods are inline oroffline. Inline de-duplication is performed by a device in the datapath. This may reduce the disk capacity required to store electronicdata thereby increasing cost savings. A disadvantage of inlinede-duplication is that the data is processed while it is beingtransmitted for backup, which may slow down the backup process.

In contrast, offline data de-duplication does not perform the datade-duplication in the data path, but instead performs the process at thebackup system. This may require more data storage capacity, such as, forexample, disk capacity. Performance may, however, be improved by havingthe process reside outside of the data path, after the backup job iscomplete. In other words, because the data is processed after beingtransmitted for backup it generally will not slow the transmission ofdata down.

In some systems, data de-duplication technology uses a dictionary basedhashing to eliminate redundant sets of variable size blocks within thedata stream. The dictionary lookup method is very effective in reducingthe data, however, this approach requires extensive processing power andfast storage devices to reduce the data. This can mean that manydictionary based de-duplication approaches are not suitable for tapebackup and may require high disk bandwidth in the Virtual Tape Librarysystems. Accordingly, in some cases it may be advantageous to usede-duplication technology that does not use dictionary basedde-duplication approaches.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed toward methods and systems for datade-duplication. More particularly, in various embodiments, the presentinvention provides systems and methods for data de-duplication that mayutilize a vectoring method for data de-duplication wherein a stream ofdata is divided into data sets or blocks. For each block, hashes, CyclicRedundancy Codes (“CRCs”) or any other code that might represent a blockmay be calculated and stored. In some embodiments, one or more codes maybe stored at the end of the blocks. The first block of the set may bewritten normally and its address and hash can be stored and noted.Subsequent block hashes may be compared with previously written blockhashes.

In accordance with some embodiments, blocks that do not match previoushashes can be written in the normal manner. Blocks with hashes thatmatch previously written blocks may be treated in a special manner whereonly the vector address (back-link) is written to indicate the locationof the “parent block.” (The previously written block with the matchingCRC or hash.) This process may eliminate redundant data. In some casesthe systems and methods described herein may perform data de-duplicationwithout the need for the large and complex directory searches.

In various embodiments, for added data integrity and performance,back-link and span can be set and special markers as often used in tapedrive for “defect skip” are written into the storage medium to indicatethe absence of de-duped block locations. The systems and methodsdescribed herein may be used for disk, Virtual Tape Libraries, tapeswhereby the vectored data is self describing, or other data storagedevices.

In some embodiments, the stream of data may be divided into data blocksthat contain user data. The blocks may be of fixed size and may be setaccording to some predetermined rules. These rules might vary fromimplementation to implementation. Various embodiments might use a hashmatch that is effective to increase the probability of data match.Another embodiment might use a sliding window to optimize the hashmatches.

In some embodiments, once the data is divided into blocks, the hash orCRC may be calculated and stored at the end of each block. In someembodiments, large CRC's might be selected to protect against“collisions.” Collisions are caused when a block has a matching CRC eventhough the underlying data does not match. The larger the CRC the lowerthe probability of a collision.

In some embodiments, when the hash or CRC match, the block hash or CRCmay be used as token for subsequent matches. In other embodiments, forexample, embodiments with shorter CRCs, blocks with matching CRCs may becompared to verify that the blocks do indeed match.

Blocks that do not have previously matched hashes or CRCs may be writtennormally, for example, like current tape drive formats. For blocks withCRC that match previously written blocks, only the “back-link” iswritten. The back-link may vector to any location using multiple blocksor block spans. The basic format of the systems and methods describedherein may be adapted to accommodate various disk formats, tape formats,and just about any other digital storage device format. Example tapeformats can include Digital Linear Tape (“DLT”), Super Digital LinearTape (“SDLT”), drives that have varying number of heads and logicaltracks including entity scrambling.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention, in accordance with one or more embodiments, isdescribed in detail with reference to the following figures. Thedrawings are provided for purposes of illustration only and merelydepict typical or example embodiments of the invention. These drawingsare provided to facilitate the reader's understanding of the inventionand shall not be considered limiting of the breadth, scope, orapplicability of the invention. It should be noted that for clarity andease of illustration these drawings are not necessarily made to scale.

FIG. 1 is a block diagram illustrating one possible configuration of anetwork that can serve as an example environment in which the presentinvention can be implemented.

FIG. 2 is a diagram illustrating an example vector de-duplication formatin accordance with various embodiments of the systems and methodsdescribed herein.

FIG. 3 is a diagram illustrating an example block “un-write” processingin accordance with various embodiments of the systems and methodsdescribed herein.

FIG. 4 is a diagram illustrating an example of block port processing inaccordance with various embodiments of the systems and methods describedherein.

FIG. 5 is a diagram illustrating an example page entry in accordancewith various embodiments of the systems and methods described herein.

FIG. 6 is a diagram illustrating an example page entry definition inaccordance with various embodiments of the systems and methods describedherein.

FIG. 7 is a diagram illustrating an example of error correction codeblock generation in accordance with various embodiments of the systemsand methods described herein.

FIG. 8 is a diagram illustrating an example of page entry and controlfield in accordance with various embodiments of the systems and methodsdescribed herein.

FIG. 9 is a diagram illustrating an example of write mode data blockprocessing in accordance with various embodiments of the systems andmethods described herein.

FIG. 10 is a diagram illustrating an example logical representation of afull size entity in cache in accordance with various embodiments of thesystems and methods described herein.

FIG. 11 is a flowchart illustrating an example method in accordance withvarious embodiments of the systems and methods described herein.

The figures are not intended to be exhaustive or to limit the inventionto the precise form disclosed. It should be understood that theinvention can be practiced with modification and alteration, and thatthe invention be limited only by the claims and the equivalents thereof.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Various embodiments of the systems and methods described herein providedata de-duplication that may utilize a vectoring method for datade-duplication. In this vectoring method for data de-duplication, astream of data may be divided into pieces, data blocks or chunks ofdata. In various embodiments, these pieces, blocks, or chunks may be thesame size. In some embodiments, the block size might be selected suchthat the pieces, data blocks or chunks are the same size as a block onthe storage media used to store the data. By selecting a block size thatis the same as the block size on the media each block will fit in theblock without left over space. Additionally, in some embodiments, theseblocks may store a vector that points to another equal sized block.

In some embodiments, a block size that does not match the size of ablock on a data storage device may be used. For example, in someembodiments, a logical block might be used. The logical blocks might bemade up of virtual blocks stored on a storage device. In suchembodiments, a block might start or end anywhere in a data storagedevice block. Additionally, in some embodiments, this may be done usingan offset. For example, an address might include a block number and ablock offset. The block number might be based on a data storage deviceblock or a virtual block. The offset may indicate where within theseblocks the start or end of a stored block occurs.

For each block, hashes, Cyclic Redundancy Codes (“CRCs”) or any othercode that might represent a block may be calculated and stored. In someembodiments, one or more hashes, CRCs, or other codes may be stored atthe end of the blocks. A hash or hash function is a reproducible methodto turning one block of data into a number that represents that block ofdata. The hash is generally smaller than the original block of data andmight serve as a digital “fingerprint” of the block of data. A CRC is atype of hash function that may be used to produce a checksum to detecterrors in transmission or storage.

The first block of the set may be written normally and its address andhash can be stored and noted. Subsequent block hashes may be comparedwith previously written block hashes. The vector address of a block witha hash or CRC that matches a previously written block may be written toindicate the location of the previously written block. In this way,redundant data may be eliminated.

In some embodiments a vector may be a pointer, an address, or otherinformation that indicate where data may be found. For example, a vectormay be a set of numbers. These numbers may be relative to a specificblock or the number may be relative to the block where the vector isstored. For example, each block might be assigned a number. The numbersmight be ordinal numbers, such as 1, 2, 3, 4, 5, etc. A vector might bestored in block 5. In one example, the vector stored in block 5 mightpoint to block 2. Some embodiments might indicate this by storing theblock number “2”. Other embodiments might use a vector that is relativeto block 5. For example, “3” might be stored because 5−3=2. Variousother vector addressing schemes might be used, as will be appreciated bythose of skill in the art.

In various embodiments, a vector might include a capsule number, avessel number, a block number and an offset. A vessel is a set ofcapsules. It may be an entire storage device, part of a storage deviceor parts of multiple storage devices. A capsule is a set of physicaldata blocks. In some cases a capsule with an error correcting code maybe referred to as an entity. In these embodiments a vessel might alsoinclude the error correcting codes for the capsules it contains. In someembodiments, a block number may indicate the physical block, logicalblock, or block on a data storage device where data begins. In variousembodiments, the data may be offset within the block. Accordingly, anoffset might be used to indicate where data begins in a block. Blocksand offsets might be used to store the start of data, the end of data,or other locations within a record or other data entity.

A record may be a logical block that can start in any physical block andend in any physical block. In other words, records may vary in size,with one record larger than another. These records, sometimes referredto as logical blocks may span several physical blocks. In someembodiments a physical block may be addressed using a start block and anoffset. In this way the physical block might start in any part of ablock on the actual data storage device.

Before describing the invention in detail, it is useful to describe anexample environment with which the invention can be implemented. FIG. 1is a block diagram illustrating one possible configuration of a networkthat can serve as an example environment in which the present inventioncan be implemented. The network might be wired or wireless. FIG. 1illustrates a data storage system 100 with which the present inventioncan be implemented. System 100 in the illustrated example includescomputing devices 105 a-b, a network 110, a server 115, an array ofstorage disks 120, and a storage area network 130. Computing devices 105a-b can any of a variety of computing devices including, for example,laptops, desktops, workstations, personal digital assistants (PDAs),handheld computing devices, or other types of computing devices.

Network 110 can be implemented using any of a variety of networkarchitectures or topologies. Such networks might include, for example,the internet, a local area network (LAN), a wide area network (WAN), aplain old telephone system (POTS), or any other suitable network orcommunications channel. In the illustrated example, computing devices105 a-b and server 115 are connected to network 110. The connection tonetwork 110 can be wireless or through a wired connection.

Server 115 can be any server system such as, for example, a conventionalstandalone file server configured to provide data services to a clientdevice such as device 105 a. Server 115 can be scalable to increasestorage capacity such as, for example, by adding storage disk array 120.Disk array 120 can be implemented as, for example, a direct-attachedstorage (DAS system). In the example architecture illustrated in FIG. 1,system 100 includes a storage pool 130, which includes switch 135, diskarray 140, router 145, and a tape server 150. Server 115, disk array120, and storage pool 130 can be implemented using one or more types ofstorage architectures such as, for example, small computer systeminterface (SCSI), serial advanced technology attachment (SATA), serialattached SCSI (SAS), or fiber channel (FC).

Generally, a legacy SCSI system with an 8-bit wide bus can typicallydeliver data at a rate of approximately 40 megabytes per second (MBps),whereas contemporary 16-bit wide bus SCSI systems can deliver data up to320 MBps. Typical SATA systems are generally less expensive than anequivalent SCSI system and can provide performance close to that of the16-bit wide bus SCSI system at 300 MBps.

FC systems offer several advantages such as pooled resources, flexiblebackup capability, scalability, fast data transfer (up to 800 MBpsfull-duplex 4 Gbit link), and the ability to accommodate long cablelengths. FC systems may have cable lengths up to 10 kilometers ascompared to a maximum cable length of 25 meters for other system suchas, for example, a SCSI system.

With continued reference to FIG. 1, the illustrated exemplary system 100can provide data access and storage redundancy by storing data atmultiple locations such as server 115, disk arrays 120 and 140, or tapeserver 150. Server 115 can be groups of remote servers; each group maybe locally or remotely connected with other groups via a network similarto network 110. As shown in FIG. 1, server 115 may access data or backupdata to disk array 140 or tape server 150 through network 110 or via adirect connection to switch 135. In this way, server 115 has theflexibility of accessing array 140 or tape server 150 via multipleconnections and thereby avoids network bottlenecks.

In various embodiments, switch 135 is an FC data switch and tape server150 is SCSI type server. In this embodiment, router 145 is configured totransfer data between a FC data bus of FC switch 135 and a SCSI bus ofSCSI tape server 150. Although a specific architecture is describedabove, components of storage pool 130 may have a different architectureor combination of architectures such as, for example, SATA, SAS, and FC.

In system 100, data redundancy can be implemented in storage pool 130 byimplementing RAID across disk array 140. Parity data needed forreconstructing a failed data sector can be distributed by a RAIDcontroller (not shown) located in storage pool 130, across array 140, orseparately to tape server 150, or across both array 140 and tape server150. In this setup, clients 105 a-b typically cannot access data storedwithin storage pool 130 network when a critical component (e.g.,motherboard, switch 135, power supply, etc.) of node 130 fails.

From time to time, the present invention is described herein in terms ofthis example environment. Description in terms of this environment isprovided to allow the various features and embodiments of the inventionto be portrayed in the context of an exemplary application. Afterreading this description, it will become apparent to one of ordinaryskill in the art how the invention can be implemented in different andalternative environments.

FIG. 2 is a diagram illustrating an example vector de-duplication formatin accordance with various embodiments of the systems and methodsdescribed herein. Referring now to FIG. 2, a diagram including a firstvessel 200 and a second vessel 202 are illustrated. Vessels 200 and 202may include some number of valid blocks, 204, 206, and 208. Some of theblocks might be duplicate blocks 210 and 212. Note that 210 mightindicate a first instance of a duplicate block. This first instance of aduplicate block 210 may occur two or more times in the rest of the data.For example, in FIG. 2, a duplicate block occurs at location 212. Atlocation 212, rather than repeat duplicate block 210, a vector 214 maybe written to indicate where to locate the data for the duplicate block212. For example, the vector would reference the location of duplicateblock 210. In this way the amount of storage space needed to storevarious data might be reduced because the vector is smaller than theblock.

In various embodiments larger files might be stored as smaller, fixedblocks of information. Each of these might be addressable. Additionally,because a larger file might not be made up of an integer number ofsmaller fixed blocks, in some embodiments one or more blocks may bepadded with additional bits. For example, the last block might be paddedwith O's, 1's, alternating O's and 1's, etc. A trade-off exists betweenblock size and probability quantity of padding bits. Larger blocks arelikely to require more padding bits, while smaller blocks may require alarger number of vectors if many of the blocks match. Additionally,larger blocks may be less likely to match. Accordingly, less memorymight be saved. Smaller blocks may be more likely to match, which mightallow for more storage savings.

The larger the block size, the more data storage space that might besaved when a back-link is stored in place of the block. In some cases,however, the larger the block size, the lower the probability of a matchin the data and, accordingly, the lower the probability that an addressmay be stored in place of a block of data. Additionally, the block sizeshould be selected such that it contains a larger number of bits thanthe number of bits in an address. If the address is as big or bigger(contains more bits) than the block size, no data storage space will besaved because just as many or more bits will be used to store theaddress.

In some embodiments, special markers may also be written into thestorage medium to indicate the absence of de-duplicated block locations.The systems and methods described herein may be used for disk, VirtualTape Libraries, tapes whereby the vectored data is self-describing, orother data storage devices.

In some embodiments, the stream of data may be divided into data sets or“blocks” that contain user data. In various embodiments, the blocks maybe fixed may be set according to some predetermined set of rules.Embodiments that use a fixed block size may address the blocks using,for example, a block count. This block count may be based on physical orvirtual blocks. These rules might vary from implementation toimplementation. Various embodiments might use a hash match that iseffective to increase the probability of data match. Another embodimentmight use a sliding window to optimize the hash matches.

In various embodiments, once the data is divided into blocks, thehashing or CRC may be calculated and stored at the end of each block.For example, a tape drive based system might use such a system. In someembodiments, large CRC's might be selected to protect against“collisions.” Collisions are caused when a block has a matching CRC eventhough the underlying data does not match. Generally, the larger the CRCthe lower the probability of a collision. In various embodiments, theprobability of a collision may be ½^(n), where n is the number of bitsin the CRC.

In some embodiments, when the hash or CRC match, the block hash or CRCmay be used as a token for subsequent matches. In other embodiments, forexample, embodiments with shorter CRCs, blocks with matching CRCs may becompared to verify that the blocks do indeed match. Blocks that do nothave previously matched hashes or CRCs may be written normally, forexample, like current tape drive formats.

In some embodiments, for blocks with a CRC that matches previouslywritten blocks, only the back-link is written. In this way, each blockmay be addressable. Various embodiments may address blocks using, forexample, a block count. In addition to a block count, an offset mightalso be used. This may allow blocks to be stored anywhere across a blockon the storage device. For example, a block might start in the middle ofsuch a block and might also cross block boundaries. By using an addressin place of an actual CRC some processing might be eliminated. Forexample, fewer processing and comparisons steps might be necessarybecause the system might go to the address saved to get the data ratherthan use the CRC to look up having to look the CRC up in a “dictionary”in order to determine the address of the block associated with the CRC.

The back-link may be a vector address to any location using multipleblocks or block spans. The basic format of the systems and methodsdescribed herein may be adapted to accommodate various disk formats,tape formats, and just about any other digital storage device format.Example tape formats can include Digital Linear Tape (“DL T”), SuperDigital Linear Tape (“SDLT”), drives have varying number of heads andlogical tracks including entity scrambling.

In some embodiments, a counter might be used to generate addresses. Forexample, a counter might count each block that is processed as part of adata stream. The counter value for each block might then be used as thephysical address of each block. In place of blocks that are not stored,an address, for example, a prior counter value might be used.Accordingly, a series of logical addresses might be generated.

For example, a series of blocks might be received. Using a counter,these blocks might be assigned addresses such as, for example, 0, 1, 2,3, 4, and 5. If blocks 1, 3, and 5 match, then the values stored mightbe value(block 0), value(block 1), value(block 2), vector(block 1),value(block 4), vector(block 1). In this way, the spaced used to storeblocks 3 and 5 might be reduced, such as, for example, when the vectorto a block uses fewer bits when compared to the block size.

In some embodiments, some number of redundant blocks might be stored. Bystoring redundant blocks of data, the data might be accessed even if astorage device fails in some way. For example, assume that blocks 0-5are stored on a backup tape. Also assume that the data in blocks 1, 3,and 5 match, as discussed above. The data values stored on the backuptape may then be value(block 0), value(block 1), value(block 2),value(block 3), value(block 4), vector(block 1 or block 3). In otherwords, values are stored in blocks 0, 1, 2, 3, and 4 and a pair ofvectors are stored in block 5. The vectors point to the blocks 1 and 3.As will be understood by those of skill in the art, the data stored inblocks 1 and 3 are redundant because the data of blocks 1 and 3 match.

Assume that the data stored in block 1 is damaged. For example, the tapemight have a defect at the location on the magnetic medium where block 1is stored. This defect might make it impossible to read the data or thedata might be corrupt such that the data that is read is incorrect. Datavalues for block 3 might be stored on the same tape. This block mightnot be damaged and may be read.

The data from block 3 is redundant with the data that was originallystored in block 1. When block 5 is read it may contain vectors to blocks1 and 3. Accordingly, based on the vectors to blocks 1 and 3 it may bedetermined that blocks 1, 3, and 5 contain the same data values. Fromthe data value of block 3, the undamaged data block, the value of blocks1, 3, and 5 can be determined.

It will be understood that, in some examples, a block might be damaged,but readable. For example, it might have incorrect data. In someembodiments, it may be possible to flag the data as possibly incorrect.It may not be possible, in some embodiments, to determine which of, forexample, two blocks is correct. In other embodiments, an odd number ofrepetitive blocks might be stored, if the data contains, for example, atleast three of a given block. If two blocks match and one does not, thematching two blocks may be assumed correct. In this way incorrect datamight be corrected.

In some embodiments, when a vector is used instead of an entire block,it may be possible to save other data in the rest of the block. Forexample, a subsequent block might be started right after the vector,rather than at the end of the unwritten block. In this way the datastorage space needed to store a given file or files might be decreased.

In some embodiments, using a counter value to store data may replaceusing a “dictionary” lookup based on CRC or hash function. For example,in a set of blocks that do not include any redundant blocks the seriesof blocks may be read in order to retrieve the stored data. Similarly,in an example where a set of blocks include some redundancy a series ofblocks or addresses may be read. The addresses may be used to determinethe data of the redundant blocks. In systems that use a CRC or hash, thedata might need to be looked up based on the CRC or hash, rather thanaddressed directly. This may require an additional step.

In various embodiments, data may be broken into logical blocks. Theseblocks may be the same size as the blocks on a storage device. Forexample, the storage space on a disk drive might be broken into a seriesof blocks. The blocks might be the same size as the logical blocks usedto store a stream of data. In such a case the logical blocks might bestored using the device's address for that block. For example, eachlogical block might start at the beginning of a block on a device andmay fill the entire block on the device. In other embodiments, however,an offset might be used. By using an offset, blocks might cross blockboundaries. This may be used when device's block size matches logicalblock size. Generally, however, when the device's block size and thelogical block size match, blocks may be stored using an entire deviceblock, rather than crossing block boundaries on the device.

In some embodiments, the device's blocks size and the logical block sizemay not be the same. In such embodiments, the logical blocks might bestored using a device block address and an offset. By using an offset,as discussed above, a block might be stored starting in a location otherthan the beginning of a device's block. In this way “virtual blocks” maybe addressable on top of the blocks of the device used to store data. Inother words, blocks that are not the same size as the blocks on astorage device may be used to store data. By using a block number and anoffset, the beginning of a block may be addressed such that the blockdoes not have to begin at the start of a block.

In some embodiments, a capsule may be used to store data. A capsule maybe a collection of physical blocks. For example, a capsule may containall physical blocks of a storage device, such as a disk drive, tapedrive, or other storage device.

In various embodiments, a vessel may include a number of physical blocksthat are the same size. A storage system may include multiple vessels.These vessels may have different physical block sizes. For example, somesystems may include different storage devices, such as disk drives, tapedrives, etc. These devices may have different block sizes. For example,the disk drive might have a different block size from the tape drive. Itwill be apparent to those of skill in the art, however, that differentdisk drives may have different block sizes from each other, differenttape drives may have different block sizes from each other and variousother data storage devices may have different block sizes from eachother.

In an embodiment that includes multiple vessels, a block may beaddressed by vessel number, physical block number and offset. In thisway, each vessel in a system may be addressed and each block within thevessels may be addressed. Additionally, by using an offset, data may notbe required to begin and end at a physical block boundary. In someembodiments, padding might also be used to fill a block.

FIG. 3 is a diagram illustrating example block “un-write” processing inaccordance with various embodiments of the systems and methods describedherein. Referring now to FIG. 3, a series of records 300 areillustrated. A record may occupy, for example, one or more blocks ofdata. In various embodiments, a block may be the smallest readable orwritable unit of data on the storage medium. In various embodimentsapproximately 12 kilobytes is used for each block, however, it will beunderstood that other block sizes are possible.

The records 300 may include matching records 302 and 304. For example,in various embodiments the matching records 302 and 304 may be indicatedby matching hashes, a CRC match, etc.

It will be understood by those of skill in the art that different CRCsor hashes may have a higher probability of indicating a matching record.In some embodiments, for example, CRCs made up of a large number of bitsmight be used. Using a large number of bits may lower the probability ofan incorrect match as compared to fewer bits because generally, as thenumber of bits increases, it becomes less likely that a block of bitswith different data will generate the same CRC. For example, a CRC with10 bits will have 2¹⁰ different possible CRC values.

In some embodiments matches might be verified. For example, when a CRCmatch occurs, the “matching” blocks might be compared bit-by-bit,byte-by-byte, word-by-word, etc., in order to determine that an actualmatch has occurred. Alternatively, in another embodiment, another CRCmight be calculated. For example, when a match occurs new CRCs might becalculated for the “matching” blocks. The new CRCs might use more bits.In this way the probability of a collision may be decreased. In anotherembodiment, the CRCs might be calculated in a different way, such thatgetting a match for two different blocks for both methods of calculatingthe CRC is extremely low, or in some cases, not possible. In someembodiments, this second check might use fewer bits because it is onlyintended to double check a previous matching CRC between the two blocks.In some embodiments, one or more of the methods discussed above mightalso be combined to find matching blocks. In various embodiments a spanCRC might be based on a narrow hash match, while a block CRC might bebased on the size of a compressed record.

FIG. 4 is a diagram illustrating an example of block port processing inaccordance with various embodiments of the systems and methods describedherein. Referring now to FIG. 4, a normal block may include 1 or morepages, each representing a record or partial record contained within theblock. FIG. 4 illustrates a block containing 3 pages representingcomplete records A 402 and B 404, and a partial record C 406 whichbegins in the block shown but ends in a subsequent block. Pagesrepresenting partial records exist because block size and record sizemight be such that a record spans two or more blocks. The block mightalso contain a table of page entries 408 describing each page of theblock. FIG. 4 illustrates page entry table 408 stored in the blockfollowing series of record pages 402, 404, and 406.

FIG. 5 is a diagram illustrating an example page entry in accordancewith various embodiments of the systems and methods described herein.Referring now to FIG. 5, a record page “J” 500 is illustrated. Invarious embodiments, after record page J 500 a record CRC 502 (“R_CRC”)may be written. The record CRC may be a code based on the uncompressedbits of an entire record, for example, record J 500. If record J 500 iscompressed, a block CRC 504 (“B CRC”) might also be stored after therecord 500. The block CRC may be a code based on the bits of an entirecompressed record. One or more of these CRCs might be used to determinewhen a de-duplicate back-link to a record of data might be used.

A data block may include all data or, in some cases filler data 506might be used to fill in any unused space between the page data and thepage entry table. In various embodiments OxOO may be used as filler data506 as illustrated in FIG. 5. For example, filler may occur when thereis not enough room between the block CRC and the page entries to justifystarting a new block. For example, in some embodiments, it might not beworth starting another block if less than three words are left. In sucha case it might be better for the next block to be part of a subsequentgroup of records. Another example of when it might not be worth startinganother block is when a flush operation forces a block to tape before itis completely full. In some embodiments, an EDC (error detection code)or CRC might be calculated over the entire block (including page data,filler, and page entry table) and be appended to the block following thepage entry table 508.

FIG. 6 is a diagram illustrating an example page entry definition inaccordance with various embodiments of the systems and methods describedherein. Referring now to FIG. 6, the page entry definition may includereserved bits. These bits may allow for future modifications to thedefinition. In various embodiments these bits are set to O. The pageentry definition may also include a compressed flag. When this bit isset it indicates that the record is compressed.

Additionally, the page entry definition may also include a record type.In various embodiments this is 3 bits and may indicate if the record is,for example, filler, record, or filemark. Bit “N” indicates that therecord continues into the next block. Similarly, bit “P” indicates thatthe record continues from the previous block. Bit “L” indicates thatthis record is the last record and page entry in a block. Someembodiments might include a page byte count, which indicates the bytesof compressed data of the record in a page if the data is compressed.The page entry definition may also include a record byte count thatindicates the uncompressed length of the record.

Some of the example embodiments with respect to the figures aredescribed here with reference to one or more tape drive back-up systems.It will be understood, however, that various systems and methodsdescribed herein may be used in conjunction with other storage devicesand systems, including, for example, hard drives, flash drives, diskdrives, etc.

An outer error correction code may be generated for a set of consecutiveblocks referred to as an entity. FIG. 7 is a diagram illustrating anexample of entity ECC block generation in accordance with variousembodiments of the systems and methods described herein. In variousembodiments, outer correction code correction blocks 702 may begenerated from the data blocks 704. For example, in various embodiments,112 data blocks 704 might generate 16 error correction blocks 702. Invarious embodiments, the 112 data blocks 704 and 16 error correctionblocks 702 may be formatted using a standard Reed-Solomon code. Asillustrated in FIG. 7, the error correction code blocks 702 mayimmediately follow the last data block 704. In various embodiments thereare no filler blocks between data 704 and error correction code 702. Asdiscussed above, in various embodiments, each data block of the 112 datablocks 704 may also have an associated “inner error correction code.” Invarious embodiments, the inner error correction code may also beappended to each outer ECC block before passing the block to the mediaaccess layer. Matches of outer ECC codes to those of preceding entitiesmay allow de-duplicate back-links replacing whole entities of data.

FIG. 8 is a diagram illustrating an example of page entry and controlfields in accordance with various embodiments of the systems and methodsdescribed herein. Referring now to FIG. 8, in various embodimentsfirmware may build a control field CFI 800 and CF2 802. The controlfields may be written to the data block. For example, in variousembodiments, the control fields may be written at the end of a 12,288byte data block, as illustrated in FIG. 8. An inner error correctioncode (or block level ECC 904) may be generated for each blockindependently, and written following the control field CF2 802. Amatching inner error correction code may be used to determine when ade-duplicate backlink to a block of data might be used.

FIG. 9 is a diagram illustrating an example of write mode data blockprocessing in accordance with various embodiments of the systems andmethods described herein. As illustrated in FIG. 9, in some embodiments,a data block 902 may be processed by a serial formatter and passed to acapsule. When the data block is passed to the capsule it may, in someembodiments, include an inner error correction code 904. In variousembodiments, a capsule might be a collection of physical blocks.

FIG. 10 is a diagram illustrating an example of a vessel de-duplicationprocess in accordance with various embodiments of the systems andmethods described herein. Referring now to FIG. 10, capsules 0 to n areillustrated. The capsules may be a collection of physical blocks.Additionally, all blocks may be the same size. In various embodiments 8entries may be included per capsule. Each entity may include 128 blocks.For example, in various embodiments the 128 blocks might include 112data blocks and 16 error correction codes.

Various embodiments might include one vessel and the physical blocksmight be the same size as the logical blocks. Accordingly, suchembodiments might not use an offset because each logical block might bestored in a data storage device block. In other embodiments, the datastorage device blocks might not be the same size as the logical blocks.Accordingly, in some embodiments, the starting point of a logical blockmight be stored as a data storage device block number and an offset fromthe beginning of the data storage device block. Additionally, the end ofa block of data might also be stored as a block number and an offset. Insome embodiments, data might be padded to fill a physical block, alogical block, or both.

Additionally, some systems might include one or more vessels. Forexample, the vessels may be disk drives, tape drives, or other datastorage devices. Accordingly, a logical data block might be stored as avessel number, physical block number, offset and a block count. In someembodiments, the block count may be the number of blocks for a givenfile, group of files, etc.

FIG. 11 is a flowchart illustrating an example method in accordance withsome embodiments of the systems and methods described herein. Referringnow to FIG. 11, in a step 1100 a code is determined. In variousembodiments, the code may, for example, be determined using a hashfunction. The code might also be a CRC.

In some embodiments, a stream of data may be divided into blocks ofdata. A hashing or other function may then be used on each block of datato determine the code for that block of data. The code, for example, ahash, CRC, etc. may be used to identify the block of data. In this wayblocks of data may be compared to each other to determine if any blocksof data match each other.

In some embodiments, the blocks of data may all be the same length.Generally, the smaller the blocks of data, the more likely it will bethat blocks will match each other. The shorter the block length,however, the less data storage space that will be saved when a matchoccurs. As discussed above, the number of bits in a block shouldgenerally be larger than the number of bits in an address so thatstoring the address actually leads to a savings in storage space. If theaddress contains more bits than the block size more bits will be neededto store the address than to store the actual data.

In a step 1102 the code may be compared to other codes to determine if ablock matches any other blocks. For example, in various embodiments,each time a code is determined for a block of data (step 1100) the codemay be stored. Additionally, when a code is determined it may becompared to previously stored codes. When codes match this may indicatethat the blocks of data match.

In some embodiments, additional steps may be taken to help avoidcollisions. A collision may occur when a code matches but the data inthe block does not actually match. For example, the blocks might becompared to each other bit-by-bit, word-by-word, etc. If the comparisondetermines that the blocks match then an address might be stored inplace of the block in a step 1104.

In another embodiment, another code, such as a hash, CRC, etc. might becalculated for the blocks that are found to be a possible match. Inother words, blocks that have matching codes may be compared usinganother code. The second code might be determined using a differentmethod or a code containing more bits. This may decrease the probabilityof a collision because the longer code might have a lower probability ofa false match.

Additionally, in some embodiments, the functions used to calculate thecodes might be selected such that when both sets of codes indicate amatch the blocks match exactly or have a very high probability ofmatching. A new code calculated for each block may be compared todetermine if the blocks match or have a greater probability of matching.If the comparison determines that the blocks match, or probably match,then an address might be stored in place of the block in a step 1104.

In the step 1104, a vector that points to a block containing the datamay be stored. In some embodiments, a counter might be used to determineaddresses as a stream of data is received and processed. For example,assume that data blocks 0-5 are to be transmitted over a communicationslink, stored on a disk drive, tape drive, etc. If blocks 2 and 5 match,then, when block 5 is processed, the match will be determined becausethe code for block 2 should match the code for block 5. In place ofblock 5 an address, such as a vector to block 2 may be stored. If theaddress uses less data storage space than the block of data would, thenthe extra data storage space might be used to save other data.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not of limitation. Likewise, the various diagrams maydepict an example architectural or other configuration for theinvention, which is done to aid in understanding the features andfunctionality that can be included in the invention. The invention isnot restricted to the illustrated example architectures orconfigurations, but the desired features can be implemented using avariety of alternative architectures and configurations. Indeed, it willbe apparent to one of skill in the art how alternative functional,logical or physical partitioning and configurations can be implementedto implement the desired features of the present invention. Also, amultitude of different constituent module names other than thosedepicted herein can be applied to the various partitions. Additionally,with regard to flow diagrams, operational descriptions and methodclaims, the order in which the steps are presented herein shall notmandate that one embodiment be implemented to perform the recitedfunctionality in the same order unless the context dictates otherwise.

Although the invention is described above in terms of various exemplaryembodiments and implementations, it should be understood that thevarious features, aspects and functionality described in one or more ofthe individual embodiments are not limited in their applicability to theparticular embodiment with which they are described, but instead can beapplied, alone or in various combinations, to one or more of the otherembodiments of the invention, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus the breadth and scope of the presentinvention should not be limited by any of the above-described exemplaryembodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more,” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

A group of items linked with the conjunction “and” should not be read asrequiring that each and every one of those items be present in thegrouping, but rather should be read as “and/or” unless expressly statedotherwise. Similarly, a group of items linked with the conjunction “or”should not be read as requiring mutual exclusivity among that group, butrather should also be read as “and/or” unless expressly statedotherwise. Furthermore, although items, elements or components of theinvention may be described or claimed in the singular, the plural iscontemplated to be within the scope thereof unless limitation to thesingular is explicitly stated.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “module” does not imply that the components or functionalitydescribed or claimed as part of the module are all configured in acommon package. Indeed, any or all of the various components of amodule, whether control logic or other components, can be combined in asingle package or separately maintained and can further be distributedacross multiple locations.

Additionally, the embodiments set forth herein are described in terms ofexemplary block diagrams, flow charts and other illustrations. As willbecome apparent to one of ordinary skill in the art after reading thisdocument, the illustrated embodiments and their various alternatives canbe implemented without confinement to the illustrated examples. Forexample, block diagrams and their accompanying description should not beconstrued as mandating a particular architecture or configuration.

What is claimed is:
 1. A computerized method for vectored datade-duplication, comprising: comparing a de-duplication code for a firstblock of data in a first vessel to a de-duplication code for apreviously processed block of data stored in a repository, where thefirst vessel stores a non-empty set of capsules, and where a capsulestores a non-empty set of blocks; and upon determining that thede-duplication code for the first block of data matches thede-duplication code for the previously processed block of data: storing,in the repository, in a location where the first block of data wouldhave been placed, a vector that includes data for locating thepreviously processed block of data; where the first vessel can berecreated from the repository without reference to other de-duplicationdata structures, and where the repository includes self-describing data.2. The method of claim 1, where the first vessel is a data storageapparatus.
 3. The method of claim 2, where the data storage apparatus isa tape, a disk, a flash drive, a solid state drive, or a storage areanetwork.
 4. The method of claim 3, where the repository is a second,different vessel.
 5. The method of claim 4, where the second vessel is asecond, different data storage apparatus.
 6. The method of claim 5,where the second data storage apparatus is a tape, a disk, a flashdrive, a solid state drive, or a storage area network.
 7. The method ofclaim 1, where the first vessel is stored on two or more data storagedevices.
 8. The method of claim 5, where the second vessel is stored ontwo or more data storage devices.
 9. The method of claim 7, where thesecond vessel is stored on two or more data storage devices.
 10. Themethod of claim 1, where the vector includes a vessel number, a blocknumber, an offset, or a count.
 11. The method of claim 1, where thevector includes a start address and an offset.
 12. The method of claim1, where the vector includes data for locating a duplicate of thepreviously processed block of data in the repository.
 13. The method ofclaim 1, comprising: upon determining that the de-duplication code forthe first block of data matches the de-duplication code for thepreviously processed block of data, verifying that the first block ofdata matches the previously processed block of data.
 14. The method ofclaim 13, where verifying that the first block of data matches thepreviously processed block of data includes performing a bit-by-bitcomparison of the first block of data and the previously processed blockof data, a byte-by-byte comparison of the first block of data and thepreviously processed block of data, or a word-by-word comparison of thefirst block of data and the previously processed block of data.
 15. Anapparatus, comprising: a processor; a memory; a set of computer hardwarecomponents that perform vector based de-duplication for a vessel ofcomputer-readable data blocks, where the vessel stores a non-empty setof capsules, and where a capsule stores a non-empty set of blocks; andan interface that connects the processor, the memory, and the set ofcomputer hardware components, where the set of computer hardwarecomponents includes: a first hardware component that identifies whethera candidate block of data in the vessel is a duplicate of a stored blockof data in the vessel by comparing a de-duplication code for thecandidate block of data and a de-duplication code for the stored blockof data; and a second hardware component that selectively replaces thecandidate block of data with a vector.
 16. The apparatus of claim 15,comprising: a third hardware component that selectively leaves up to athreshold number of redundant copies of the candidate block of data inthe vessel.
 17. The apparatus of claim 15, where the vessel is stored ontwo or more data storage devices.
 18. The apparatus of claim 17, wherethe two or more data storage devices include a tape, a disk, a flashdrive, a solid state drive, or a storage area network.
 19. The apparatusof claim 15, where the vector includes a vessel number, a block number,an offset, or a count.
 20. The apparatus of claim 15, where the vectorincludes data for locating a duplicate of the stored block of data inthe vessel.